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In 2010-11, Wake County Public School System 
(WCPSS) was in district-wide improvement as a 
result of failing to meet Adequate Yearly Progress 
(AYP) in mathematics at the district level for the 
second consecutive year. Under the No Child Left 
Behind Act of 2001 (NCLB), the district was 
required to develop a Local Educational Agency 
(LEA) Plan for Improvement that incorporated 
scientifically-based research strategies and to offer 
training appropriate to address the subgroups that did 
not make AYP. District Improvement requires 10% 
of Title I funds be set aside for professional 
development. 

WCPSS has implemented several approaches focused 
on providing greater training support to the subject 
areas and student groups that need it most. 

• Sheltered Instruction Observation Protocol 
(SIOP®), a research-based approach aimed at 
strengthening students’ academic language and 
student involvement, is one of the strategies which 
have been in place within WCPSS since 2007-08. 

• Secondary Mathematics and Secondary Literacy 
efforts are more recent additions to District 
Improvement training efforts. 

This report examines overall student outcomes 
related to these three strategies in 2010-1 1 as well as 
overall teacher outcomes and longitudinal results for 
schools targeted for three consecutive years of SIOP® 
training and support compared to matched schools 
who were not involved. A report on the 
implementation of WCPSS’ District Improvement 
efforts in 2010-11 was released recently and should 
be considered a companion document 
in understanding the results found 
(Bulgakov-Cooke & Baenen, 2011). 

WAKE COUNTY 

PUBLIC SCHOOL SYSTEM 



Major Findings 

Study Design 

Two levels of analysis were conducted to assess 
student outcomes: 1) overall results for SIOP®, 
Secondary Literacy, and Secondary Mathematics 
compared to WCPSS; and 2) an examination of 
SIOP® schools receiving three years of support and 
matched schools. 

School Characteristics 

In 2010-11, six elementary schools and three 
middle schools received additional training and 
support for the implementation of SIOP®. Twelve 
schools were supported by the Secondary Literacy 
initiative and 19 by Secondary Mathematics. 

SIOP® 

Slop® schools had a greater increase in students 
reaching growth targets than in the district overall. 
Increases in reading and mathematics proficiency 
at SIOP® elementary schools and reading at SIOP® 
middle schools were similar or slightly higher than 
for WCPSS. Overall mathematics results were 
not positive for SIOP® middle schools; however, 
matched school analysis by subject and school 
level found targeted student subgroups (i.e., 
Hispanic/Latino and Black/ African American , 
Limited English Proficient, and economically 
disadvantaged) did benefit from attending SIOP® 
targeted schools. 

Secondary Literacy 

The percentage of students meeting growth targets 
increased more than the district. The increase in 
students proficient in reading at schools supported 
by Secondary Literacy was similar to WCPSS. 

Secondary Mathematics 

Growth was lower for participating schools than 
for WCPSS overall. The increase in students 
proficient in mathematics at schools supported by 
Secondary Mathematics was similar to WCPSS. 

Recommendations 

SIOP efforts should continue in schools with 
high concentrations of student subgroups not 
making AYP, and other District Improvement 
components should increase this focus. Secondary 
Literacy and Secondary Mathematics initiatives 
should strengthen implementation levels and 
consistency, which in turn should increase the 
likelihood of stronger achievement outcomes in 
their second year. 


The author acknowledges the contributions made by WCPSS’ Federal Programs and Grants, Academics, 
and Data and Accountability staff to the data collection and analysis presented in this report. 
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STUDY DESIGN 

This report examined 2010-11 overall outcomes related to the SIOP®, Secondary Literacy, and 
Secondary Mathematics efforts implemented within WCPSS’ District Improvement Plan. The 
objective of the District Improvement Plan, through the implementation of these initiatives, was 
to increase reading and mathematics achievement for the targeted NCLB subgroups in 
elementary, middle, and high schools. The ultimate goal is for WCPSS to meet AYP targets for 
two consecutive years and exit district improvement. The key questions of interest in this 
evaluation were to assess whether District Improvement interventions are helping WCPSS make 
progress towards this long-term goal. 

• Have schools targeted for SIOP®, Secondary Literacy, and Secondary Mathematics been 
effective in terms of outcomes for students (compared to the district overall)? 

• Did SIOP® targeted schools and teachers with the highest implementation ratings show 
greater student growth? 

• How did the students in SIOP® targeted schools progress compared to non-targeted (matched 
schools)? 

• How did the students in SIOP® targeted schools progress by NCLB subgroups? 

In order to address these questions two levels of analysis were conducted: 

1 . Overall student results on End of Grade/End of Course tests (EOG/EOC) for SIOP®, 
Secondary Literacy, and Secondary Mathematics compared to WCPSS; and 

2. An examination of student outcomes on EOG for SIOP® schools receiving three years of 
support and matched schools constructed from non-targeted schools. 

In addition to considering overall proficiency, growth, and AYP results based on state EOG tests 
for the schools receiving extra support implementing SIOP®, Secondary Literacy, and Secondary 
Mathematics, a three-year change in the percentage of students meeting growth was calculated. 
The three-year change was calculated as the difference between the percentage of students 
meeting growth in 2008-09 and that of 20 1 0- 1 1 . 1 These values were calculated for: 

• District-wide elementary schools 

• Selected SIOP® elementary schools within the district 

• District-wide middle schools 

• Selected SIOP® middle schools within the district 

For each of these four groups of schools, the three-year change was also calculated for the 
following subgroups of students: Black/ African American, Hispanic/Latino, White, economically 


1 For each year, the percentage of students meeting growth was calculated as the total number of students meeting 
growth divided by the total number of students considered to be available for growth measurement by virtue of 
having a previous year’s growth value and being enrolled at their school for the requisite number of school days. 
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disadvantaged (ED), limited English proficient (LEP) students, and students with disabilities 
(SWD). In a few cases certain subgroups were too small (i.e., fewer than 15 students) to have 
their growth data reported in the disaggregated growth spreadsheets. In such cases, the total 
values used for the calculations were produced without the unreported subgroup. 

Comparisons to matched schools were conducted to improve the strength of the evaluation by 
including only schools that had received three years of additional support implementing SIOP® 
and schools matched on achievement and demographic characteristics (the matched process is 
described in the next section). These analyses represent the most precise test of SIOP® 
effectiveness for students’ growth. 

SCHOOL CHARACTERISTICS 

The characteristics of schools targeted for additional support in 2010-1 1 under the District 
Improvement Plan are presented first, followed by the schools which received three years of 
additional support implementing SIOP® and matched schools. 

District Improvement Targeted Schools 

The overall results of three major District Improvement components — SIOP®, Secondary 
Literacy, and Secondary Mathematics — were compared to WCPSS achievement results overall. 
The schools that received targeted support with each component in 2010-1 1 were examined to 
determine the effectiveness of each initiative. 

The first portion of the study focused on the impact of 2010-1 1 services among schools receiving 
additional support in implementing SIOP®’ the Secondary Literacy Initiative, and the Secondary 
Mathematics Initiative. 

• Six elementary schools — Brentwood, Durant Road, Fox Road, Hodge Road, Timber Drive, 
and Wilburn — and three middle schools — East Wake, West Millbrook, and Zebulon — 
received additional training and support with the implementation of SIOP®. 

• The Secondary Literacy Team provided support to English/Language Arts teachers at 12 
schools targeted to receive support based on achievement needs. 

• The Secondary Mathematics Team provided additional support to grade 6-8 mathematics 
teachers and Algebra I teachers at 19 schools with the goal of helping support their students 
who performed below grade level and who did not meet AYP targets. 
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SIOP® Schools Receiving Support for Three Years and Matched Schools 

The second portion of the study was focused on SIOP® schools receiving support for three years 
and matched schools. Although nine schools received additional training and support for the 
implementation of SIOP® in 2010-1 1, for Hodge Road Elementary, 2010-11 represented the 
second year of receiving additional support. The remainder of schools were in their third year of 
receiving support; therefore, to be consistent and to examine the full impact of implementation, 
Hodge Road was excluded from the matched school analysis. 

The eight SIOP® schools for which 2010-11 was their third year receiving support were the focus 
of the second part of this study: five elementary schools — Brentwood, Durant Road, Fox Road, 
Timber Drive, and Wilburn — and three middle schools — East Wake, West Millbrook, and 
Zebulon. Cluster analysis was conducted to select five matched elementary schools and three 
middle schools. The analysis was run using the centroid method: 2 four variables were included 
in the model (2009-10 performance composite, 2010-11 overall risk score, 2010-11 days in 
membership on the 20th day of school, and school level). The overall risk score is a school level 
score calculated based on the percentage of students at each school with academic risk factors, 
such as ED status, LEP status, and SWD. 

Table 1 displays the eight schools for which 2010-11 was their third year receiving support 
implementing SIOP® and their matched schools. Schools receiving support with implementing 
but with less than three years of support and schools that had received school- wide training on 
SIOP® were excluded from the analysis and thus not utilized as possible matches. 

Table 1 

Three Year SIOP® Schools and Matched Schools 


SIOP® School 

Matched School 

Elementary Schools 

Brentwood 

Barwell 

Fox Road 

Millbrook 

Durant Road 

Willow Springs 

Timber Drive 

Forestville Road 

Wilburn 

Lynn Road 

Middle Schools 

East Wake 

East Millbrook 

West Millbrook 

Fuquay-Varina 

Zebulon 

Carroll 


Data Source: Cluster analysis utilizing the centroid method 
conducted using the 2010-11 school data file 
from WCPSS’ Growth and Planning Department. 


2 “ Centroid method. The cluster to be merged is the one with the smallest sum of distances between cluster means 
(centroids) for all variables. The centroid method also weights for differences in cluster size” (Garson, 2010). 
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The next section examines the characteristics of the students included within this evaluation and 
is organized to reflect the two levels of analysis: 1) the characteristics of students enrolled in 
schools receiving support implementing SIOP®, Secondary Literacy, and Secondary 
Mathematics components of the District Improvement Plan compared to WCPSS; and 2) the 
characteristics of the students enrolled in the SIOP® schools receiving three years of support and 
matched schools. 

STUDENT CHARACTERISTICS 

Among SIOP® schools, ED, LEP, Black/ African American, and Hispanic/Latino student 
subgroups were disproportionately represented. 

District Improvement Targeted Schools 

As Table 2 shows, the nine schools which received additional support implementing SIOP® in 
2010-11 had some subgroups of students who were disproportionately represented. The 
overrepresentation of these subgroups, which failed to make AYP standards, was done 
intentionally. 3 

• ED students represented nearly two-thirds (62. 1%) of elementary school students and more 
than half (53.6%) of middle school students attending SIOP® schools compared to just over a 
third in the district (36.9% and 35.8% respectively). 

• LEP students represented 21.5% of elementary students attending SIOP® schools compared 
to 1 1.8% in the district and 9.0% of middle school students compared to 5.7% of the district. 

• Black/ African American elementary students represented 37.2% versus 23.1% of WCPSS 
students and 34.6% middle school students compared to 26.6% in the district. 

• Hispanic/Latino elementary students represented 28.8% versus 16.5% of WCPSS students 
and 21.2% of middle school students compared to 13.3% in the district. 


3 2010-1 1 targeted schools were selected based on 2009-10 data. In 2009-10, WCPSS failed to meet AYP in 
mathematics for Black/ African American students and SWD students at the high school level, Hispanic/Latino 
students at the middle school level, and Black/African American and ED students at the elementary school level. 
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Table 2 

Characteristics of Students Attending SIOP® Schools, 2010-11 



Students 

Attending 

SIOP® 

Elementary 

Schools 

WCPSS 

Elementary 

Students 

Students 

Attending 

SIOP® 

Middle 

Schools 

WCPSS 

Middle 

Students 


Number 

Percent 

Number 

Percent 

Number 

Percent 

Number 

Percent 

ED 

2,756 

62.1% 

25,601 

36.9% 

1,335 

53.6% 

11,717 

35.8% 

SWD 

531 

12.0% 

7,968 

1 1 .5% 

425 

17.1% 

4,682 

14.3% 

LEP 

953 

21 .5% 

8,159 

1 1 .8% 

224 

9.0% 

1,880 

5.7% 

Male 

2,277 

51 .3% 

35,315 

51 .0% 

1,303 

52.3% 

16,595 

50.7% 

Female 

2,161 

48.7% 

33,993 

49.0% 

1,189 

47.7% 

16,115 

49.3% 

American Indian 

17 

0.4% 

253 

0.4% 

19 

0.8% 

154 

0.5% 

Asian 

133 

3.0% 

4,553 

6.6% 

46 

1 .8% 

1,984 

6.1% 

Black/ African Am. 

1,650 

37.2% 

16,027 

23.1% 

863 

34.6% 

8,715 

26.6% 

Hispanic/Latino 

1,277 

28.8% 

1 1 ,468 

16.5% 

528 

21.2% 

4,362 

13.3% 

Multiracial 

197 

4.4% 

3,093 

4.5% 

95 

3.8% 

1,437 

4.4% 

White 

1,164 

26.2% 

33,914 

48.9% 

941 

37.8% 

16,058 

49.1% 

Total 

4,438 

100% 

69,308 

100% 

2,492 

100% 

32,710 

100% 


Note: Students will appear in more than one category: race and gender, ED, SWD, and/or LEP. 

Data Source: 2010-11 WCPSS End-of-Year Elementary and Middle School Student Rosters and WCPSS 
Demographics: School Statistics and Maps, 2010-11 at 
http://www.wcpss.net/demographics/reports/book09a.pdf 
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As Table 3 shows, the schools which received support through the Secondary Literacy and 
Secondary Mathematics Initiatives in 2010-11 were similar demographically to elementary and 
middle (i.e. secondary) students in WCPSS overall. The only two subgroups over represented 
were ED students and Black/ African American students within targeted schools. 


Table 3 

Characteristics of Students Attending Schools Targeted for Secondary Literacy and/or 

Secondary Mathematics, 2010-11 



Students 
Attending 
Secondary 
Literacy Schools 

Students 

Attending 

Secondary 

Mathematics 

Schools 

WCPSS 

Secondary 

Students 


Number 

Percent 

Number 

Percent 

Number 

Percent 

ED 

5,182 

39.6% 

8,873 

32.5% 

23,083 

31.9% 

SWD 

1,716 

13.1% 

3,477 

12.7% 

9,195 

12.7% 

LEP 

682 

5.2% 

1,343 

4.9% 

3,574 

4.9% 

Male 

6,733 

51.5% 

14,044 

51 .4% 

36,978 

51.1% 

Female 

6,337 

48.5% 

13,255 

48.6% 

35,406 

48.9% 

American Indian 

63 

0.5% 

125 

0.5% 

332 

0.5% 

Asian 

333 

2 . 5 % 

1,460 

5.3% 

4,264 

5.9% 

Black/ African American 

4,589 

35.1% 

7,484 

27.4% 

19,157 

26.5% 

Hispanic/Latino 

1,943 

14.9% 

3,441 

12.6% 

9,042 

12.5% 

Multiracial 

579 

4.4% 

1,245 

4.6% 

3,180 

4.4% 

White 

5,563 

42.6% 

13,544 

49.6% 

36,409 

50.3% 

Total 

13,070 

100% 

27,299 

100% 

72,384 

100% 


Note: 1. Secondary schools include students attending middle and high school. 

2. Students will appear in more than one category: race and gender, ED, SWD, and/or LEP. 
Data Source: 2010-11 WCPSS End-of-Year Middle and High School Student Rosters 
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MATCHED GROUP DEMOGRAPHICS 

The student demographics at the matched schools are depicted in Table 4. While Table 2 
displays the demographic characteristics of all students attending SIOP® schools in 2010-1 1, 
Table 4 represents only schools targeted for three consecutive years of SIOP® training and 
support and the comparison schools that were matched to them. One SIOP® school was 
excluded from the matched analysis because they were involved in SIOP® less than three years. 
Dropping this school had little impact on the overall demographic patterns for the SIOP® group. 

Table 4 shows the demographic characteristics of the 3,685 elementary school students and 
2,492 middle school students attending schools receiving three years of support implementing 
SIOP®. While matching was conducted at the school level rather than individual one-to-one 
student matches, the distribution of the demographic characteristics of students attending SIOP® 
schools with three years of support and matched schools were very similar. 


Table 4 

2010-11 Characteristics of Students Attending SIOP® Schools 
Receiving 3 Years of Support 



SIC 
Eleme 
School 
3 Yer 
Sup] 

>P® 
:ntary 
s with 
irs of 
port 

Matched 

Elementary 

School 

Students 

SIC 
Mid 
School 
3 Yer 
Sup] 

>P® 
Idle 
s with 
irs of 
port 

Matched 
Middle School 
Students 


Number 

Percent 

Number 

Percent 

Number 

Percent 

Number 

Percent 

ED 

2,197 

59.6% 

2,099 

56.3% 

1,335 

53.6% 

1,439 

52.7% 

SWD 

451 

12.2% 

424 

1 1 .4% 

425 

17.1% 

471 

17.3% 

LEP 

688 

18.7% 

531 

14.2% 

224 

9.0% 

199 

7.3% 

Male 

1,875 

50.9% 

1,926 

51 .7% 

1,303 

52.3% 

1,368 

50.1% 

Female 

1,810 

49.1% 

1,802 

48.3% 

1,189 

47.7% 

1,362 

49.9% 

American Indian 

17 

0.5% 

7 

0.2% 

19 

0.8% 

12 

0.4% 

Asian 

119 

3.2% 

58 

1 .6% 

46 

1 .8% 

67 

2.5% 

Black/ African Am. 

1,393 

37.8% 

1,454 

39.0% 

863 

34.6% 

1,083 

39.7% 

Hispanic/Latino 

907 

24.6% 

828 

22.2% 

528 

21 .2% 

469 

17.2% 

Multiracial 

173 

4.7% 

205 

5.5% 

95 

3.8% 

138 

5.1% 

White 

1,076 

29.2% 

1,176 

31 .5% 

941 

37.8% 

961 

35.2% 

Total 

3,685 

100% 

3,728 

100% 

2,492 

100% 

2,730 

100% 


Note: Students will appear in more than one category: race and gender, ED, SWD, and/or LEP. 

Data Source: 2010-11 WCPSS End-of-Year Elementary and Middle School Student Rosters and WCPSS Demographics: 
School Statistics and Maps, 2010-11 at http://www.wcpss.net/demographics/reports/book09a.pdf 
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ACADEMIC ACHIEVEMENT RESULTS 

Question 1: Have schools targeted for SIOP®, Secondary Literacy, and Secondary Mathematics 
been effective in terms of outcomes for students (compared to the district overall)? 

The percentage of students proficient in reading and mathematics at schools targeted for support 
implementing SIOP®, Secondary Literacy Initiative, or Secondary Mathematics Initiative all 
showed improvement in the percentage of students scoring at grade level (proficient) in reading 
and mathematics, but increases were either similar or only slightly higher than for WCPSS 
overall. Students attending schools supported by SIOP® (with the exception of middle school 
mathematics) and Secondary Literacy experienced greater growth than the district; this was not 
true among schools supported by the Secondary Mathematics Initiative. 

Academic Proficiency 

Figure 1 displays the percentage of students in grades 3-5 proficient in reading and mathematics 
for SIOP® targeted schools and WCPSS overall. From 2008-09 to 2010-1 1, there was an 
increase in the percentage of students proficient in reading and mathematics at targeted schools 
and for WCPSS overall. From 2008-09 to 2010-1 1 SIOP® targeted schools had fewer students 
proficient than for WCPSS overall. While both SIOP® schools and WCPSS experienced a 
percentage point increase, the increase in the percentage of students proficient in reading and 
mathematics at the targeted schools was just slightly higher (1 to 2 percentage points) than 
WCPSS overall. 


Figure 1 

Percentage of Students in Grades 3-5 Proficient, 2008-09 to 2010-11 



Data Source: 2010-1 1 WCPSS Elementary and Middle Disaggregated Charts 
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Figure 2 displays the percentage of students in grades 6-8 proficient in reading and mathematics 
at SIOP® targeted schools, schools implementing Secondary Literacy and Secondary 
Mathematics Initiatives, and WCPSS overall. For all groups considered there was an increased 
percentage of students proficient in reading and mathematics from 2008-09 to 2010-1 1. While 
schools targeted for SIOP® had fewer students proficient in 2008-09 and 2010-11 than WCPSS 
overall, the increase in the percentage of students proficient in reading and mathematics at the 
targeted schools was slightly higher than seen in WCPSS overall (by three to four more 
percentage points). Little difference was evident for Secondary Literacy and Secondary 
Mathematics Initiatives versus the system between 2009-10 (prior to implementation) and 2010- 
1 1 (the first year of implementation). 


Figure 2 

Percentage of Students in Grades 6-8 Proficient 
2008-09 to 2010-11 



Note: MS = middle school 

Data Source: 2010-1 1 WCPSS Disaggregated Charts for Elementary and Middle Schools. 

Academic Growth 

Another way to gauge success in improving achievement is to examine the percentage of 
students reaching annual state growth targets. Examining the percentage of students reaching 
growth is a more sensitive method of assessing student gains even when growth was not 
sufficient to change level scores. The state’s ABCs growth formula reflects approximately one 
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year’s growth for one year of instruction for each student. Schools are considered to show high 
growth if 60% of their students reach their growth target. 

Elementary growth results were quite positive. The percentage of students in grades 4 and 5 who 
met growth standards increased from 2008-09 to 2010-1 1 at SIOP® targeted schools and WCPSS 
overall (see Figure 3). Indeed, increases at SIOP® targeted schools were 10 percentage points in 
reading and 1 1 percentage points in mathematics. These percentage point increases were more 
than double that experienced by WCPSS overall (3.5 percentage point increase in reading and a 
4.5 percentage point increase in mathematics above increases in WCPSS overall). In 2010-1 1, 
students enrolled in the SIOP® targeted elementary schools met high growth in reading and 
mathematics, with 60.0% and 66.0% of students meeting their growth targets respectively. 


Figure 3 

Percent of Students in Grades 4 and 5 Meeting Growth 
2008-09 to 2010-11 



Data Source: 2010-1 1 WCPSS Elementary and Middle School Disaggregated Charts 


The results in grades 6-8 were mixed; reading results were positive but math results were not. 

• In reading, the percentage of students in grades 6-8 who met growth increased from 2008-09 
to 2010-1 1 (see Figure 4) for all three groups compared, with the district increasing less than 
the SIOP targeted and the literacy initiative schools. For SIOP® targeted middle schools, the 
percentage increased nine points between 2008-09 and 2009-10 and then dropped by two 
percentage points between 2009-10 and 2010-11, with a net gain of seven points across two 
years. The Secondary Literacy Initiative actually began in 2010-1 1, so the impact that might 
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be attributed to the initiative between 2009-10 and 2010-11 was 1.5 percentage points. The 
district overall increased 4.5 percentage points over two years, with almost all of this 
occurring between 2008-09 and 2009-10. 


• In mathematics, however, results for the two math initiatives were not favorable compared to 
the district. For SIOP® math, the percentage of students meeting growth standards increased 
slightly between 2008-09 and 2009-10 and then decreased between 2009-10 and 2010-11 (for 
a net loss of three percentage points). Middle school math efforts began in 2010-1 1; the 
percentage of students meeting growth targets declined five points between 2009-10 (before 
service) and 2010-11 (after service). The district stayed stable between 2008-09 and 2009- 
10, and declined only 0.3 percentage points between 2009-10 and 2010-11. 


Figure 4 

Percentage of Students in Grades 6-8 Meeting Growth 
2008-09 to 2010-11 



Data Source: 2010-1 1 WCPSS Elementary and Middle School Disaggregated Charts 


Three- Year Change in Students Meeting Growth Targets 

Figures 5-8 display the three-year percentage point changes 
in students meeting growth targets by ethnicity and academic 
risk factor and are focused on SIOP® targeted elementary 
and middle schools. 


r > 

For SWD students in reading and 
LEP students in mathematics, the 
three-year percentage point 
increases in students meeting growth 
was considerably higher than the 
district. 
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Reading 

At the elementary level, both WCPSS overall and SIOP® targeted schools experienced a positive 
percentage point change in students meeting reading growth targets. This percentage point 
change at SIOP® schools was even greater than for WCPSS overall for each subgroup of interest. 
In elementary school, SWD and Hispanic/Latino student subgroups not only had the largest 
percentage point change from 2008-09 to 2010-1 1, but also had the largest difference between 
their subgroups and the WCPSS gains overall. 

Figure 5 

Three- Year Percentage Point Change in Students Meeting Reading Growth Targets 

2008-09 to 2010-11, Grades 4-5 



Note: This figure displays the percentage point difference between the percentages of students meeting 
growth targets in 2009-09 to those meeting growth targets in 2010-1 1 . 


Number of Students Associated with Three- Year Percentage Point Change in Figure 5 

2008-09 to 2010-11, Grades 4-5 




All 

Black/ 

African 

American 

Hispanic/ 

Latino 

ED 

LEP 

SWD 

White 

WCPSS 

2008-09 

10,764/18,356 

1,838/3,631 

463/915 

2,393/4,802 

69/137 

452/939 

6,129/9,850 

2010-11 

11,856/19,517 

2,328/4,210 

1,569/2,632 

3,453/6,174 

359/633 

872/1,536 

6,308/10,076 

SIOP® 

2008-09 

667/1,257 

219/464 

115/235 

287/612 

35/72 

35/91 

263/419 

2010-11 

730/1,214 

249/460 

194/289 

407/679 

87/141 

44/70 

229/362 


Note: Numerator = the percentage of students meeting growth, Denominator = total group. 

Data Source: 2010-11 WCPSS Elementary School Disaggregated Charts 
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For middle schools, as for elementary schools, both WCPSS overall and SIOP® targeted schools 
experienced a positive percentage point change in students meeting reading growth targets. The 
percentage point change at SIOP® schools was greater than for WCPSS overall and for all 
subgroups considered with the exception of White students; thus, promoting the closing of 
achievement gaps. Indeed, each middle school subgroup considered (with the exception of 
White students) had a change in students meeting reading growth targets greater than 10 
percentage points from 2008-09 to 2010-1 1. 


Figure 6 

Three- Year Percentage Point Change in Students Meeting Reading Growth Targets 

2008-09 to 2010-11, Grades 6-8 



Note: This figure displays the percentage point difference between the percentages of students meeting 
growth targets in 2009-09 to those meeting growth targets in 2010-1 1 . 


Number of Students Associated with Three- Year Percentage Point Change in Figure 6 
2008-09 to 2010-11, Grades 6-8 




All 

Black/ 

African 

American 

Hispanic/ 

Latino 

ED 

LEP 

SWD 

White 

WCPSS 

2008-09 

14,279/26,336 

3,200/6,694 

1,324/2,629 

3,453/7,240 

836/1,695 

1,583/3,324 

8,054/14,240 

2010-11 

16,231/27,659 

3,749/6,659 

2,014/3,369 

4,756/8,405 

681/1,134 

1,735/3,028 

8,505/14,478 

SIOP® 

2008-09 

1,092/2,294 

353/839 

183/387 

442/1,049 

107/239 

126/330 

462/905 

2010-11 

1,120/2,033 

370/652 

252/413 

564/989 

88/144 

112/231 

430/839 


Data Source: 2010-11 WCPSS Middle School Disaggregated Charts 

Note: Numerator = the percentage of students meeting growth, Denominator = total group. 
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Mathematics 


SIOP® targeted elementary schools experienced a positive percentage point change in students 
meeting mathematics growth targets overall and for each subgroup considered. Additionally, the 
percentage point change at SIOP® schools was greater than WCPSS overall and for the 
subgroups considered with the exception of the SWD subgroup. One highlight is that, while 
WCPSS overall had a decrease of 1 1 percentage points for growth between 2008-09 and 2010- 
1 1, the LEP student subgroup at SIOP® targeted elementary schools had a one percentage point 
increase. 


Figure 7 

Three-Year Percentage Point Change in Students Meeting Mathematics Growth Targets 

2008-09 to 2010-11, Grades 4-5 


15 


S3 

a 


, j,i 


-5 


-10 


6 SIOP Schools: 

Brentwood 
Durant Elementary 
Fox Road 
Hodge Road 
Timber Drive 
Wilburn 




-15 

All 

Black 

Hispanic 

ED 

LEP 

SWD 

White 

■WCPSS 

4.1 

1.4 

1.0 

0.9 

-11.2 

5.4 

6.2 

SIOP® 

8.4 

7.1 

2.8 

5.4 

1.2 

1.9 

13.8 


Note: 1. Percentage point scale ranges from -15 to 15 in order to capture negative results. 

2. This figure displays the percentage point difference between the percentages of students 
meeting growth targets in 2009-09 to those meeting growth targets in 2010-1 1. 


Number of Students Associated with Three-Year Percentage Point Change in Figure 7 
2008-09 to 2010-11, Grades 4-5 




All 

Black/ 

African 

American 

Hispanic/ 

Latino 

ED 

LEP 

SWD 

White 

WCPSS 

2008-09 

11,866/18,476 

2,197/673 

606/925 

3036/4927 

157/204 

531/989 

6,400/9,885 

2010-11 

13,445/19,683 

2,595/240 

1793/2695 

3919/6267 

448/681 

985/1,668 

7,185/10,121 

SIOP® 

2008-09 

739/1,279 

257/476 

154/241 

359/624 

73/105 

52/91 

246/423 

2010-11 

813/1,228 

284/465 

198/297 

432/687 

104/147 

52/88 

262/364 


Note: Numerator = the percentage of students meeting growth, Denominator = total group. 

Data Source: 2010-11 WCPSS Elementary School Disaggregated Charts 
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At the middle school level, WCPSS and SIOP® targeted middle schools either remained constant 
or experienced a percentage point decrease in students meeting mathematics growth targets from 
2008-09 to 2010-1 1 overall and for each subgroup considered (with the exception of WCPSS’ 
Black/ African American and SWD student subgroups). 


Figure 8 

Three- Year Percentage Point Change in Students Meeting Mathematics Growth Targets 

2008-09 to 2010-11, Grades 6-8 



Note: 1. Percentage point scale ranges from -15 to 15 in order to capture negative results. 

2. This figure displays the percentage point difference between the percentages of students 
meeting growth targets in 2009-09 to those meeting growth targets in 2010-1 1. 


Number of Students Associated with Three- Year Percentage Point Change in Figure 8 

2008-09 to 2010-11, Grades 6-8 




All 

Black/ 

African 

American 

Hispanic/ 

Latino 

ED 

LEP 

SWD 

White 

WCPSS 

2008-09 

16,454/26,402 

3,714/6,688 

1,609/2,660 

4,083/7,279 

1,091/1,753 

1,786/3,326 

9,164/14,255 

2010-11 

17,252/27,808 

3,753/6,683 

1,996/3,439 

4,702/8,511 

765/1,258 

1,745/3,068 

9,301/14,495 

SIOP® 

2008-09 

1,376/2,298 

477/838 

239/391 

620/1,053 

153/245 

176/326 

554/904 

2010-11 

1,178/2,051 

369/658 

239/423 

563/1,007 

98/157 

128/238 

484/838 


Note: Numerator = the percentage of students meeting growth, Denominator = total group. 

Data Source: 2010-11 WCPSS Middle School Disaggregated Charts 
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Adequate Yearly Progress (AYP) 

Given WCPSS was in district-wide improvement as a result of failing to meet AYP in 
mathematics at the district level, this section presents the AYP results for schools targeted by 
district improvement efforts and WCPSS overall. A long-term goal of district-wide improvement 
efforts was to succeed in exiting this status. This goal became considerably more difficult in 
2010-11, due to a considerable increase in AYP targets in 2010-1 1, as the 100% targets set for 
2013-14 approaches. The targets increased 28.4 percentage points in reading and 11.4 
percentage points in mathematics for elementary and middle schools; and 30.8 and 15.8 
percentage points (in reading and mathematics respectively) for high schools. Thus, these results 
should be considered within the context of the higher targets required to meet AYP. With the 
new targets in place, each grade span — elementary, middle, and high school — experienced a 
decrease in the percentage of AYP targets met for WCPSS overall and each District Improvement 
initiative considered (see Figures 9-11). 


Figure 9 

K-5 Percentage of AYP Targets Met 
2009-10 and 2010-11 
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Note: 

Data Source: 


Number of schools included in figure are: In 2009-10, WCPSS = 102, SIOP® =14, and Math 
Coaches = 44; and in 2010-11, WCPSS = 103, SIOP® = 7, and Math Coaches = 52 
2009-10 and 2010-11 AYP by Level: Elementary, Middle, and High 
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Figure 10 

Grades 6-8 Percentage of AYP Targets Met 
2009-10 and 2010-11 



Note: Number of schools included in figure are: In 2009-10, WCPSS = 30, SIOP® = 5, Secondary 

Math = 7, and Secondary Literacy = 2; and in 2010-1 1, WCPSS = 33, SIOP® = 6, Secondary 
Math = 7, and Secondary Literacy = 2 

Data Source: 2009-10 and 2010-11 AYP by Level: Elementary, Middle, and High 
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Figure 11 

Grades 9-12 Percentage of AYP Targets Met 
2009-10 and 2010-11 
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Note: Number of schools included in figure are: In 2009-10, WCPSS = 23, Secondary Math =11, 

and Secondary Literacy =10; and in 2010-11, WCPSS = 25, Secondary Math =11, and 
Secondary Literacy =10 

Data Source: 2009-10 and 2010-11 AYP by Level: Elementary, Middle, and High 


Question 2: Did SIOP® targeted schools and teachers with the highest implementation ratings 
show greater student growth? 

In the spring of 2011, Praxis Research, Inc. staff collaborated with the Data and Accountability 
(formally known as Evaluation and Research) and District Improvement staff to modify a 
research-based protocol; originally generated by the Center for Applied Linguistics (CAL) and 
customized for WCPSS. 4 Praxis staff, along with WCPSS’ District Improvement coaches, 
monitored the implementation of SIOP® at 15 WCPSS elementary and middle schools. A 
stratified random sample of mathematics, language arts, and English as a Second Language 
(ESL) classrooms was selected. Additional observations from a convenience sample of available 
teachers supplemented the sample. Eight monitors conducted 171 observations ranging from 30 


4 The Center for Applied Linguistics worked with WCPSS staff to customize the existing research-based SIOP® 
protocol (this instrument was utilized to observe, rate, and provide feedback on lessons) and verily the inter-rater 
reliability of the revised instrument (Echevarria, Vogt, & Short, 2008). 
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minutes to an hour in order to determine the level of SIOP® implementation occurring within 
each classroom. Utilizing a SIOP® Observation Summary Sheet, they monitored for lesson 
preparation, building background, comprehensible input, strategies, interaction, practice and 
application, lesson delivery, and review and assessment. Within each of these areas several 
features (26 in all) were rated using a 1-5 scale in which 5 indicated high implementation. The 
26 ratings were averaged to create an overall implementation score for that observation. 

Targeted SIOP® Schools 

Table 5 displays the average SIOP® implementation score and the average 2010-11 reading and 
mathematics academic change (AC) score by school. The state ABCs AC score reflects whether 
students as a group grew more or less than the target projection for that year. A growth score of 
zero means the target was met exactly. Table 5, which is sorted by SIOP® implementation score, 
shows in general that schools with higher SIOP® implementation scores also received higher 
reading and mathematics AC scores. It also illustrates that five of the eight schools with SIOP® 
targeted support for three or more years had higher implementation and AC scores (but three did 
not). 


Table 5 

Average of SIOP® Implementation Score and Mean 
2010-11 AC Score of Targeted SIOP® Schools 


School 

Average SIOP® 
Implementation 
Score 

Average 

2010-11 AC Score 

Reading 

Math 

Harris Creek Elementary 

2.83 

0.17 

0.28 

Westlake Elementary 

3.09 

0.07 

0.11 

Zebulon Middle* 

3.11 

0.02 

0.04 

W. Millbrook Middle* 

3.12 

0.04 

0.02 

N. Garner Middle 

3.13 

0.09 

0.09 

E. Garner Middle 

3.21 

0.13 

0.05 

Wilburn Elementary* 

3.24 

-0.02 

0.09 

Combs Elementary 

3.33 

0.11 

0.10 

E. Wake Middle* 

3.42 

0.11 

0.15 

Fuquay-Varina Elementary 

3.42 

0.17 

0.43 

Durant Road Elementary* 

3.45 

0.11 

0.27 

Timber Drive Elementary* 

3.61 

0.20 

0.22 

Wakefield Elementary 

3.61 

0.08 

0.18 

Fox Road Elementary* 

3.66 

0.06 

0.20 

Brentwood Elementary* 

3.70 

0.18 

0.25 

Total 

3.30 

0.09 

0.13 


Note: 1. The SIOP® implementation score ranged from 1-5. 

2. Bolded Blue font indicates schools with the average 2010-11 AC scores above the 
overall average. 

3. * indicates SIOP® schools receiving three years of support with implementation. 

Data Source: 2010-11 WCPSS End-of-Year Elementary and Middle School Student Rosters and 

2010-11 SIOP® observation data 
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Teachers Implementing SIOP® 

In order to further investigate the relationship between fidelity of implementation and student 
achievement, this section examines the relationship between the Teacher Effect Education Value 
Added Assessment System (EVAAS®) scores and SIOP® implementation scores. Teacher Effect 
scores from EVAAS®, which are based on EOG scores; estimate a teacher’s influence on 
students’ academic progress. The Teacher Effect score is a function of the difference between 
students’ predicted and actual scores; thus, a 0.0 represents an average teacher. 

SIOP® observations each resulted in an implementation score of 1-5. Teachers with more than 
one score due to multiple observations were assigned one score based on the average of their 
scores. 


EVAAS scores are generated for teachers teaching grade 5-8; thus, this analysis represents only a 
segment of the K-8 teachers observed. Based on the 40 teachers with available EVAAS® and 
SIOP® observation data, teachers with high implementation scores (above the mean score of 
3.30) had a higher percentage with a positive Teacher Effect EVAAS® score, although the 
percentage did not reach statistical significance. A positive Teacher Effect EVAAS® score 
indicates a teacher who was able to move their students on average above what was predicted. 

MATCHED SCHOOL ACHIEVEMENT RESULTS FOR SIOP® 


In order to improve the strength of the evaluation, a matched school analysis was conducted. 
Student achievement outcomes were expected for students once SIOP® was in a school for three 
years. Because the program was rolled out over time, we conducted a matched school analysis 
restricted to schools that had received three years of additional support implementing SIOP® and 
schools matched on achievement and demographic characteristics (the results of the matched 
analysis address the study’s next question). 


Question 3: How did the students in SIOP® targeted schools progress compared to non-targeted 
(matched schools)? 


From 2008-09 to 2010-1 1, SIOP® schools had a greater decrease in 
students with a Reading and Mathematics EOG score of Level I and 
a greater increase in students scoring Level IV than did matched 
schools (with the exception of middle school students with a 
Mathematic EOG score of Level I). 5 SIOP® middle schools did, 
however, experience a significant decrease in students scoring Level 
II on their mathematics EOG which was not seen at matched 
schools. SIOP® and matched schools both had a significant increase 
in EOG Reading Level III students from 2008-09 to 2010-1 1. This 


•N 

A higher percentage of 
students attending 
SIOP ® middle schools 
were able to move up to 
Level IV in reading and 
mathematics between 
2009-10 and 2010-11 
than were students at 
matched schools. 

J 


5 EOG Levels III and IV represent proficiency. 
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difference was similar at the elementary school level, but SIOP® middle schools experienced a 
larger increased than did matched schools. 


Academic Proficiency 

The 2008-09 reading and mathematics EOG proficiency levels for students who attended schools 
with three years of support implementing SIOP® and students at matched schools are presented 
in Figures 12 and 13. The 2008-09 EOG scores were used to capture student performance prior 
to the examination of 2010-1 1 EOG results. Overall proficiency rates (Level III and IV) for the 
two groups were similar at both the elementary and middle school levels (varying < five 
percentage points by subject and school level). 


Figure 12 

2008-09 Reading and Mathematics EOG Level for 
SIOP® and Matched Elementary Schools 
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Data Source: 2008-09 WCPSS Winscan files 

Interpretation Example: 16.6% of students attending schools targeted for SIOP® support for three years scored a 
Level IV on their reading EOG in 2008-09 compared to 18.0% of students at matched 
schools. 
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Figure 13 

2008-09 Reading and Mathematics EOG Level for 
SIOP® and Matched Middle Schools 
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Data Source: 2008-09 WCPSS Winscan files 


Tables 6 and 7 display the 2008-09 and 2010-11 EOG levels for schools targeted for three years 

of support implementing SIOP® and matched schools. Elementary and middle school reading 

results revealed: 

• SIOP® and matched schools both had a significant decrease in Level I students from 2008-09 
to 2010-1 1; however, the decrease was larger among SIOP® schools. 

• SIOP® and matched schools both had a significant increase in Level III students from 2008- 
09 to 2010-1 1. This difference was similar at the elementary school level, but SIOP® middle 
schools experienced a larger increased than did matched schools. 

Mathematics results at SIOP® and matched schools revealed: 

• SIOP® elementary schools had a significant decrease in Level I students from 2008-09 to 
2010-11 while for matched schools there was a 0.1 percentage point increase. 

• SIOP® and matched middle schools, both had a significant decrease in Level I students from 
2008-09 to 2010-1 1; however, the decrease was larger among matched schools. 

• SIOP® middle schools had a significant decrease in Level II students and a significant 
increase in Level IV students from 2008-09 to 2010-1 1. 

• Matched middle schools had a significant increase in Level III students from 2008-09 to 
2010 - 11 . 
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Table 6 

Reading and Mathematics EOG Levels for Elementary Students Attending 
3- Year SIOP® Schools and Matched Schools 



Reading 

Math 

2008-09 

2010-11 

Change 

2008-09 

2010-11 

Change 

Level I 

SIOP® 

18.0% 

13.5% 

■4.5%* 

5.2% 

2.8% 

-2.4%* 


Matched 

13.1% 

10.4% 

-2.7%* 

2.6% 

2.8% 

0.1% 

Level II 

SIOP® 

20.5% 

20.6% 

0.2% 

18.7% 

18.1% 

-0.6% 


Matched 

20.3% 

20.3% 

-0.1% 

16.0% 

15.7% 

-0.3% 

Level III 

SIOP® 

44.9% 

49.0% 

4.1%* 

55.4% 

54.2% 

-1.2% 


Matched 

48.6% 

52.8% 

4.2%* 

55.8% 

54.8% 

-1.0% 

Level IV 

SIOP® 

16.6% 

1 6.8% 

0.3% 

20.7% 

24.9% 

4.2%* 


Matched 

18.0% 

1 6.6% 

-1 .5% 

25.6% 

26.8% 

1 .2% 


Note: 1. * indicates the difference between 2008-09 and 2010-11 was significant based on a z statistic calculated for 

students attending SIOP® targeted schools and matched schools. 

2. Percentages under Change column are rounded to the nearest tenths place. 

Data Source: 2008-09 and 2010-11 WCPSS Winscan files 

Interpretation Example: Among students attending SIOP® targeted elementary schools that received three years of additional 
support, there was a significant increase (4.2%) in students scoring Level IV on their Mathematics EOG from 
2008-09 to 2010-11. 


Table 7 

Reading and Mathematics EOG Levels for Middle School Students Attending 
3- Year SIOP® Schools and Matched Schools 



Reading 

Math 

2008-09 

2010-11 

Change 

2008-09 

2010-11 

Change 

Level I 

SIOP® 

16.6% 

1 1 .3% 

-5.3%* 

5.4% 

4.2% 

-1 .2%* 


Matched 

15.3% 

13.1% 

-2.2%* 

7.7% 

5.5% 

-2.2%* 

Level II 

SIOP® 

22.2% 

22.2% 

0.0% 

20.8% 

16.7% 

-4.0%* 


Matched 

21 .3% 

21.8% 

0.6% 

20.4% 

21 .7% 

1 .3% 

Level III 

SIOP® 

46.1% 

50.4% 

4.3%* 

53.1% 

55.0% 

1 .9% 


Matched 

46.7% 

49.8% 

3.0%* 

51 .8% 

54.1% 

2.3%* 

Level IV 

SIOP® 

15.1% 

16.1% 

1 .0% 

20.7% 

24.1% 

3.4%* 


Matched 

16.7% 

1 5.3% 

-1 .4% 

20.1% 

18.7% 

-1.4% 


Note: 1. * indicates the difference between 2008-09 and 2010-11 was significant based on a z statistic calculated for 

students attending SIOP® targeted schools and matched schools. 

2. Percentages under Change column are rounded to the nearest tenths place. 

Data Source: 2008-09 and 2010-11 WCPSS Winscan files 
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Academic Growth of Matched Students 

Increasing the percentage of students reaching growth targets is one way to gauge success in 
improving achievement, and is more sensitive to student gains even when growth was not 
sufficient to change level scores. The state’s ABCs growth formula reflects approximately one 
year’s growth for one year of instruction for each student. Schools are considered to show high 
growth if 60% of their students reach their growth target. 

Overall Growth 

A higher percentage of students attending SIOP® schools met growth 
in reading and mathematics than did students attending matched 
schools; this difference was only significant for mathematics at the 
middle school level. Results were not significant for middle school 
reading or for elementary reading and mathematics (see Figure 14). 


A significantly higher 
percentage of students 
attending SIOP ® middle 
schools met growth in 
mathematics than did 
students attending 
matched schools. 

C > 


Figure 14 

Percentage of Students Meeting 2010-11 Reading and Mathematics 

EOG Growth Targets 



Note: * indicates a significant difference between 3-Year SIOP® schools and matched schools 

Data Source: 2010-11 WCPSS End-of-Year Elementary and Middle School Student Rosters 


Academic Change 

Academic change is another way of consider students’ growth from 2009-10 to 2010-1 1. Keep 
in mind that a growth score of zero means the target was met exactly. Box plots were used to 
depict the mean, median, and range of the AC scores for students in SIOP® targeted schools and 
students attending matched schools in 2010-1 1 (Figures 15-18). The box represents the majority 
of student scores (25 th to 75 th percentile). The “whiskers,” or vertical lines, extending from the 
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box, represent the range of scores, with the most extreme scores denoted by small boxes. The 
range of academic change scores for students attending SIOP® and matched schools by subject 
and school are illustrated in Figures 15-18. The mean and range (denoted by the “whiskers”) of 
AC scores for reading and mathematics were similar for the two groups. Within each box, the 
mean is signified by a plus sign and the median by a horizontal line in the middle of the box. 
The average academic change scores in reading and mathematics hovered close to zero, 
indicating performance close to what was expected for both SIOP® and matched schools. 


Figure 15 

Reading EOG Academic Change Score for Elementary School Students 
Attending 3- Year SIOP® Schools and Matched Schools 
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Data Source: 2010-11 WCPSS End-of-Year Elementary School Student Roster 
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Figure 16 

Reading EOG Academic Change Score for Middle School Students 
Attending 3- Year SIOP® Schools and Matched Schools 
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Data Source: 2010-11 WCPSS End-of-Year Middle School Student Roster 
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Figure 17 

Mathematics EOG Academic Change Score for Elementary School Students 
Attending 3- Year SIOP® Schools and Matched Schools 
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Note: Box Width varies with N 

Data Source: 2010-11 WCPSS End-of-Year Elementary School Student Roster 
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Figure 18 

Mathematics EOG Academic Change Score for Middle School Students 
Attending 3- Year SIOP® Schools and Matched Schools 
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Data Source: 2010-11 WCPSS End-of-Year Middle School Student Roster 
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Question 4: How did the students in SIOP® 


targeted schools progress by NCLB subgroups? 

r > 


At the elementary level, attending SIOP targeted schools seemed 
to benefit Hispanic/Latino students in reading the most. At the 
middle school level, Black/ African American and ED students 
benefited in reading and mathematics while Asian and LEP 
students benefited in mathematics. Hispanic/Latino participants 
experienced the most growth, meeting high growth (more than 
60% of student having met their growth target) in reading at 
elementary schools and reading and mathematics at middle 
schools. Among elementary school students attending SIOP 
targeted schools, all racial groups had high growth for mathematics and Hispanic/Latino and 
White students met high growth in reading. 


SIOP® targeted 
schools seemed to 
benefit Hispanic/Latino 
elementary school 
students and 
Black/African 
American, ED, and 
LEP middle school 
students the most. 


i® 


Growth by NCLB Subgroups 

The percentage of students meeting growth was examined in order to assess students’ progress 
by NCLB subgroups. Figure 19 displays the percentage of students attending 3-Year SIOP® and 
matched elementary schools who met 2010-11 reading and mathematics EOG growth target by 
racial group. 

Figure 19 

Percentage of Students Meeting 2010-11 EOG Growth Targets by Racial Group 
at 3- Year SIOP® and Matched Elementary Schools 



Note: * indicates a significant difference between 3 -Year SIOP® schools and matched schools 

Data Source: 2010-11 WCPSS End-of-Year Elementary School Student Roster 
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Figure 20 displays the percentage of students attending 3- Year SIOP® and matched middle 
schools who met 2010-11 reading and mathematics EOG growth target by racial group. A 
significantly higher percentage of Black/ African American students met growth targets in 
reading and mathematics and Asian students met those in mathematics than did matched 
students. The results for Asian students are based on a small number of students and should 
therefore be considered cautiously. Although reading results for Hispanic/Latino students were 
not significantly higher in 3 -Year SIOP® schools than for students at matched schools, it should 
be noted that at 3 -Year SIOP® schools this group of students met high growth. 


Figure 20 

Percentage of Students at 3- Year SIOP® and Matched Middle Schools Meeting 
2010-11 EOG Growth Targets by Subject and Racial Group 
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Note: * indicates a significant difference between 3 -Year SIOP® schools and matched schools 

Data Source: 2010-11 WCPSS End-of-Year Middle School Student Roster 


Figure 21 compares the percentage of students attending 3-Year SIOP® and matched schools who 
met 2010-11 EOG growth targets by subject and academic risk factor. Among academic risk 
groups there were no significant differences between the percentage of students who met growth 
targets at 3-Year SIOP® and matched elementary schools. ED and LEP students attending 3- 
Year SIOP® and matched elementary schools met high growth in mathematics and were within 
one percentage point of meeting high growth for reading. Among middle school students 
attending 3- Year SIOP® schools a significantly higher percentage of ED students met growth in 
reading and mathematics and LEP students in mathematics than did matched students. 
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Figure 21 

Percentage of Students at 3- Year SIOP® Schools and Matched Schools 
Meeting 2010-11 EOG Growth Targets by Risk Factor 
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Data Source: 


* indicates 3 -Year SIOP® schools had a significantly higher percentage of students meet growth than did matched 
schools 

2010-11 WCPSS End-of-Year Elementary and Middle School Student Rosters 


CONCLUSIONS 

This report examined 2010-11 overall outcomes related to the SIOP®, the Secondary Literacy 
Initiative, and the Secondary Mathematics Initiative implemented within WCPSS’ District 
Improvement Plan and a matched school analysis of schools for which 2010-11 represented their 
third year implementing SIOP®. The four key questions of interest in this evaluation and the 
findings related to each are presented in Table 8. 
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Table 8 

Summary Analysis of Key Research Questions 


Key Questions 

Summary of Findings 

Have schools targeted for SIOP®, 
Secondary Literacy, and Secondary 
Mathematics been effective in terms of 
outcomes for students (compared to 
the district overall)? 

SIOP® 

At SIOP® targeted elementary schools reading and mathematics proficiency 
rates were similar to WCPSS overall. However, students attending SIOP® 
schools experienced greater growth than the district (with the exception of 
middle school mathematics). Moreover, examinations of the three-year 
percentage point change by subgroup revealed that for some subgroups the 
percentage point increase was considerably higher. 

Among middle school students attending SIOP® targeted schools reading 
results were positive, with percentage point improvements in proficiency 
and growth (approximately three percentage points) above that experienced 
in the district overall. Overall the increase in the percentage of students 
proficient in mathematics was greater than for the district; however, the 
percentage of students meeting growth targets was lower than WCPSS. 
Secondary Literacy Initiative 

Among students attending secondary literacy schools the increase in students 
proficient in reading was similar to WCPSS while the percentage of students 
meeting growth increased 7.6 percentage points more than the district. 

Secondary Mathematics Initiative 

While there was an increase in the percentage of students proficient in 
mathematics at schools targeted for secondary mathematics, improvements 
were similar to WCPSS overall and growth was 3.7 percentage points lower 
than the district. 

Did SIOP® targeted schools and 
teachers with the highest 
implementation ratings show greater 
student growth? 

In general, schools with higher SIOP® implementation scores also received 
higher reading and mathematics academic change (growth) scores. Based on 
the 40 teachers with available EVAAS® and observation data, teachers with 
high implementation scores (above the mean score or 3.30) had a higher 
percentage of students making growth, although the percentage did not reach 
statistical significance. 

How did the students in SIOP® 
targeted schools progress compared to 
non- targeted (matched schools)? 

A significantly higher percentage of students attending SIOP® middle 
schools were able to move up one or more EOG levels in reading and 
mathematics between 2009-10 and 2010-11 than did matched students. 
Furthermore, a significantly higher percentage of students attending SIOP® 
middle schools met growth in mathematics than did students attending 
matched schools. 

How did the students in SIOP® 
targeted schools progress by NCLB 
subgroups? 

A significantly higher percentage of Hispanic/Latino students who attended 
3-Year SIOP® targeted elementary schools met growth in reading than did 
students who attended matched schools. Indeed, among elementary school 
students attending SIOP® targeted schools, all racial groups had high growth 
for mathematics and Hispanic/Latino and White students met high growth in 
reading. 

Among students at 3-Year SIOP® middle schools, a significantly higher 
percentage of Black/African American and ED met growth in reading and 
mathematics and a significantly higher percentage of LEP students met 
growth in mathematics than students at matched schools. 
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Overall SIOP® reading results were positive. Comparisons to the system showed greater 
increases in students reaching growth targets in SIOP® schools. In addition, some targeted 
subgroups improved more than matched students in both elementary and middle schools. Overall 
SIOP® mathematics results were not as positive as for reading. Matched school analysis, on the 
other hand, revealed stronger outcome patterns for Black/ African American, Asian, ED, and LEP 
students. 

The results for schools implementing the Secondary Literacy and Mathematics Initiatives were 
not strong; however, given these results represent the first year of implementation they should be 
used as a baseline for future examinations of student outcomes. The schools implementing the 
Secondary Literacy and Mathematics Initiatives experienced increases in students proficient in 
reading and mathematics similar to WCPSS. While schools implementing the Secondary 
Literacy Initiative experienced increases in growth greater than the district, schools 
implementing the Secondary Mathematics Initiative experienced growth lower than the district. 

Table 9 


Summary 2010-11 Achievement Outcomes by District Improvement Initiative 


Initiative 

School 

Level 

Overall 

Matched 

Overall 

NCLB Subgroups 
Growth 

Proficiency 

Growth 

Proficiency 

Growth 

SIOP® 

Reading 

Elementary 

Up slightly 
more than 
WCPSS 

> than 
WCPSS 

Up but 
similar to 
matched 

Up but 
similar to 
matched 

Hispanic students > 
than matched* 

Middle 

> than 
WCPSS 

> than 
WCPSS 

> than 
matched 

Up but 
similar to 
matched 

Black/African Am 
and 

ED students > than 
matched* 

SIOP® 

Math 

Elementary 

Up but 
similar to 
WCPSS 

> than 
WCPSS 

> than 
matched 

Up but 
similar to 
matched 

No significant 
difference 

Middle 

> than 
WCPSS 

< than 
WCPSS 

> than 
matched 

> than 
matched * 

Black/African Am, 
Asian, LEP, and 
ED students > than 
matched* 

Secondary 

Literacy 

Middle/ 

High 

Up but 
similar to 
WCPSS 

> than 
WCPSS 

N/A 

N/A 

N/A 

Secondary 

Math 

Middle/ 

High 

Up but 
similar to 
WCPSS 

< than 
WCPSS 

N/A 

N/A 

N/A 


Note: * Indicates significant difference 
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DISCUSSION 

Of the three District Improvement efforts, SIOP® has been in place the longest and has the 
strongest implementation (Bulgakov-Cooke & Baenen, 2011). Analysis of SIOP® targeted 
elementary and middle schools achievement trends revealed that increases in proficiency rates 
between 2008-09 and 2010-11 were either similar to or only slightly higher than those found in 
WCPSS overall. Analysis of growth revealed larger increases in the percentage of students who 
met growth attending SIOP® elementary schools for both reading and mathematics and SIOP® 
middle schools in reading. Thus, overall analysis indicated SIOP® implementation has not yet 
assisted these schools in closing overall proficiency gaps. However, greater growth (at the 
elementary school level growth was more than twice that seen within WCPSS overall) reflects 
improved student achievement which if continued should eventually lead to increased 
proficiency levels and eroded achievement gaps. 

While three-year analysis of student subgroups showed mathematics growth at WCPSS middle 
schools was down, and even more so for SIOP® middle schools, 2010-11 subgroup analysis of 
matched schools revealed SIOP® middle schools had a significantly higher percentage of 
Black/ African American, ED, and LEP students meeting growth than did matched schools. 
Indeed, although comparisons to WCPSS overall were not favorable, matched analysis which 
examined student achievement by prior EOG level and NCLB subgroups found positive results 
in mathematics among SIOP® targeted middle schools. Furthermore, the percentage of students 
meeting growth was significantly higher at SIOP® middle schools than matched schools. 

2010-11 represented the first year of implementation of the Secondary Literacy and Secondary 
Mathematics Initiatives. The implementation study found a small number of teachers were 
trained in 2010-1 1 and training varied considerably by school (Bulgakov-Cooke & Baenen, 
2011). Initial student outcomes for schools implementing Secondary Literacy were found to be 
either similar or more positive than the district. Given results for SIOP® schools were more 
positive than for schools implementing Secondary Literacy, and SIOP® appeared to focus on 
schools with greater numbers of students in the targeted subgroups, targeting schools with high 
concentrations of subgroups failing to make AYP at the district level may strengthen the results. 
Furthermore, the implementation study showed uneven application of secondary mathematics 
modules; thus, it should not be surprising that student achievement results were either equal or 
lower than the district (Bulgakov-Cooke & Baenen, 2011). The demographic similarity of 
schools implementing Secondary Mathematics with WCPSS coupled with the weak results 
indicates, as with Secondary Literacy, the results of this initiative could be strengthened by more 
selective process of targeting schools to receive training. 

Given that District Improvement efforts were designed to target support to schools with high 
concentrations of student subgroups which did not make AYP at the district level, the fact that 
matched school analysis revealed several of these student subgroups (Black/ African American, 
ED, and LEP middle school students) significantly outperformed similar students at matched 
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schools suggests continuation of SIOP® would benefit these student groups and improve the 
district’s chances of meeting AYP. On the other hand, the AYP standards are so high at this 
point that training efforts alone may have to be strategically paired with other efforts if AYP is 
still considered realistically obtainable. The District Improvement Plan and its initiatives have 
been employed to help WCPSS exit district-wide improvement status. However, the periodic 
increase of AYP targets as we approach the 100% targets set for 2013-14 has made it 
increasingly difficult for our system to make AYP and exit district-wide improvement. 

RECOMMENDATIONS 

The improved growth among schools targeted for support under District Improvement should 
result in improved proficiency, although this may take time. Based on results from the 
implementation study and this study’s findings, we have the following recommendations: 

• Select schools with high concentrations of student subgroups not making AYP to receive 
support implementing SIOP®’ Secondary Literacy, and Secondary Mathematics Initiatives. 
Matched school analysis of revealed subgroups of students (ED, LEP, Black/ African 
American, and LEP) performed significantly better than students at matched schools (varying 
by school level and subject). These student subgroups were among those not making AYP at 
the district level (also varying by school level and subject). The fact that these student 
subgroups preformed significantly better at SIOP® schools than at matched schools suggests 
that the selection of schools with high concentrations of students within the subgroups failing 
to make AYP may positively impact these subgroups at the district level and has the potential 
to decrease achievement gaps. Moreover, given results for SIOP® schools were more 
positive than for schools implementing Secondary Literacy and Secondary mathematics and 
schools implementing Secondary Literacy and Secondary Mathematics were 
demographically similar to the district, schools with high concentrations of student subgroups 
not making AYP should be targeted to receive support implementing these initiatives. 

• Expand and provide consistent training for the Secondary Literacy Initiative. Initial 
student outcomes were either similar or more positive than the district. The implementation 
study found a small number of teachers were trained in 2010-1 1 and training varied 
considerably by school; thus, consistency should be improved and training expanded. 

• Implement the Secondary Mathematics Initiative more consistently and on a larger scale. 
The implementation study showed uneven application of secondary mathematics modules 
and student achievement results were either equal or lower than the district. Therefore, the 
implementation of this initiative should first be strengthened by more consistent application 
of modules and outcomes reevaluated. 
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• Reexamine student outcomes for the Secondary Literacy and Secondary Mathematics 
Initiatives. 2010-11 represented the first year of implementation of these initiatives. 
Evaluation of student outcomes at the end of the first year of implementation can be 
premature. We were also not able to analyze results just for students in the classes of trained 
teachers, which would have been a more precise measure of impact. Thus, the outcome data 
presented in this report, while informative, should be considered as an initial baseline and 
further examination of student outcomes should be conducted over time. This will require 
tighter data collection on teachers and students directly impacted by the initiative. By the end 
of 201 1-12, an increase in proficiency and AYP results is expected by the initiatives. 

The status of NCLB reauthorization and waivers should be watched closely because it will have 
a major impact on our District Improvement efforts. Student results for 201 1-12 will help 
determine what we will decide to continue if we are no longer required to set aside a certain 
percentage of District Improvement funds for professional development. 
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